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The National Assessment Planning Project conducted under the auspices of the 
Council of Chief State School Officers (CCSSO) was created by a consortium of 
eighteen national organizations interested in education and in exploring the feasibility 
of expanding the National Assessment of Educational Progress (NAEP) in order to 
produce state-by-state comparisons of student achievement. 

The project is governed by a Steering Committee (Appendix A), each member 
appomted on the recommendation of an organization in the consortium. ITais report 
was conveyed to the Department of Education (ED) and to the National Assessment of 
Educational Progress on the review and approval of the Steering Committee. The 
publication, however, does not necessarily reflect the views of each of the associations 
m the consortium. 

The project was supported by Grant No. SPA-1549 from the National Science 
Foundation (NSEJ with fimds partly from NSF and partly from the Department of 
Education through an inter-^ovemmental tiransfer from ED to NSF. The mutual 
interest of the two agencies m this project and their willingness to provide joint support 
made the project possible. This puolication, however, does not necessarily reflect the 
views of either agency. 
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I. INTRODUCTION 
PURPOSE/fflSTORICAL PERSPECTIVE 

For almost twenty years, student achievement in the United States has been 
measured by the- National Assessment of Educational Progress (NAEP) and reported in 
recent years as "The Nation's Report Card." Based on a sample size of 25,000 students 
at each of three grade levels, the test has provided the only data that reflect in a 
comprehensive way what all students in tfie U.S. know and can do in various subject 
areas. It has provided descriptive information about student strengths and weaknesses 
m basic and hi^er order skills; it has provided data comparing groups of students by 
race and ethnicity, gender, type of community and region; it has provided trend data 
reflecting the ups and downs of performance over the years; it has reported positive 
correlations between achievement and some student experience variables, hi 1990, a 
large number of states will be able to participate in a state-level assessment that will 
provide "state report cards" which make possible state-by-state or state-to-nation 
comparisons of student achievement. 

Interest in state level comparisons is rising in many quarters in the belief that 
better state-level information and better state-comparative mformation about student 
achievement than is currently available would fadhtate the improvement of education 
in the individual states. In 1984 a majority of CCSSO members supported the 
development of a system of student assessment that would provide state comparisons, 
and in 1985, the members endorsed the expansion of NAEP as the most feasiole way of 
providing these conffiarisons. During the i986 national assessment, two individual 
states, Wyoming and Georgia, contracted with NAEP to conduct an in-state concurrent 
assessment and to provide state-to-nation comparisons. Also in 1986, and 1987, groups 
of southern states, m a project coordinated by the Southern Region Education Board, 
contracted- with NAEP to conduct state level assessments. They were provided reports 
comparii^ achievement among those states. Some- of the rising interest can be 
attributed to governors. A 1987 report from the National Governors' Association 
entitled Results in Education presented a number of .comparative education indicators 
and displayed a blank column for achievement, clearly expressing the intent to fill in 
that coluffla in future years with achievement data. In a 1987 report, a national group 
appointed by Secretaiy of Education William Bennett and chaired by former Governor 
of Tennessee, Lamar Alexander, made a series of recommendations on the future of 
NAEP. A major recommendation was that NAEP should be expanded to provide state- 
by-state comparisons. 

This rising interest is not without its critics. Some are worried that Federal, state 
and local poliiymakers may misuse the data, making inappropriate inferences and 
drawing unwarranted cause and effect conclusions. Fears are expressed that the test 
will be veiy influential, and with that influence, foster a national curriculum. Still 
others fear that the compromises that might be made on objectives vdll result in an 
assessment that measures the least common denominator and discourages badly needed 
curriculum reform. 

Designing a national assessment that would not only be constructive but also 
minimize potential disadvantages is the purpose of this National Assessment Planning 
Project. The project is to make recommenaations that answer two questions of major 
interest to state and local educators and policymakers, who at some time prior to 1990, 
will be asked whether they want a report card for their state. 

They will likely want to know what mathematics objectives (knowledge, skills) 
the assessment will measure and whether the objectives are more or less compatible 
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with what they believe the schools m their sutes are tivini! to teach or h»T,»m u 
Wi. " '°Pi« of > separate pubHcatiofOTSemafo ObfeS Se 

be reiSTd oS S ISf^*'^ ?P^7*^^ ^AEP. how comparisons should 
DC reported, and on several policy issues related to conducting state-level assessmente 

PROCESS FOR DF!VFTr>PTVr, RErQMMFlvp ^ T TA|tfff 

Krn.H PSPu°^!f ^^^^ resulted in this loport was designed principally to result in a 
broad, field-based consensus on the issues o?expanding to stl?e 

SrS'NteSPth^'l-^^^^^^ ^« over 
wouKeSi?^^? J^nSn?* Objectives or knowledge and skills that an asse^mem 
• consensus process is respons ve to the Federal law creatinBthe 

s^atemfn^S^t;^ '"^"^"^ f "each7eaming area aLe^mem shffi^^^^ 
statements devised through a national consensus approval * 

5,nH c»™!Sf *® recommendations in this report address analysis and reportiue matters 

NAEpfAB^^^^J^'p^,?"^^^ Po^i<^«s and approving repom\Sh ^ 

tol^JIa^fS^r ^""^^ Committee. Nevertheless, an e^ly detlmSation was made 
to engage many pohcymakers, practitioners and scholars from across thec3trvinpn 
of con3uItati6n and coUaboration. TO det?rSon w^^Sd o^ 
the behef that such a process was likely to identify significant problems w^Sh d^fmSfg 
thf nlrSfn" ^ J*" ^ constnic&ve ideas. wffiws^L^a rS 

?roi^^1?ee±?S!^^^^ '^^^^^^ approved bySe 

projects oteenng Committee, many others were mvolveA particularlv the diflctnn nf 

T^'^Z^V^^ f °^ ^^^^t^- ^^any of these SSs S 
sS^S '"^^ P'^^""' "^"y also responded to S^Shelpful 

foUowiS'fwSS^ "^""^ ^"^^ ^^^"^ ^^^^ 1988, has the 

the 18 tti^^r^^^^"" (Appendix A) with members recommended by each of 

r^nnrt "^c^^P^^ ^®P9^ Committee (Appendix B) was created to draft this 
sShS^T^S^r^'^^^^ local dis^rt SSent 

Dre£S?r^ Wnff ^ -/^^^^ of assessment The Committee met once for 
KtrSJn^-^^ ^° co^ider its charge and determine what information it wished to 
have prior to its major work-session in December. For that major work-sessioiTSe 
Committee was given maity documents, including the foUowSgf ^ 

Copies of correspondence from state directors of assessment raising issues and 
suggestmg ways that comparative data should be r^rtS ^ 



An options p^r entitled "Alternative Ways of Reporting State-by-State 
Comparisons" prepared for the Committee under a contract. 

Several recent publications on education indicators from CCSSO, the RAND 
Corporation, and others; lists of common student background questions from a 
variety of studies, including the National Education Longitudinal Study of 1988, 
Schools and Staffing Survey, Common Core of Data, High School and Beyond; 
and the publication The Underachi eving Ciirriaihim (Tnternatinnal Study of 
Math Achievement). 

A background memorandum from the Southern Regional Education Board 
reviewing the procedures followed in eiglit southern states, in which school 
district employees administered NAEP tests. 

A report provided by the National Association of State, Directors of Special 
Education on the inclusion of special students in NAEP. 

During its December meeting, the Committee made tentative decisions about 
the recommendations it wished to make. In subsequent weeks, a report was drafted 
based on those decisions and copies were sent to state education agencies (Appendix 
C), and selected state policymakers, local district educators and scholars (Appendix D). 
Project staff discussed this report with the assessment subcommittee of the Committee 
on Evaluation and Information Systems (CEIS) of the Council of Chief State School 
Officers in a special meeting in January. 

All comments received on the draft wereprovided to the Analysis and Reports 
Committee which met again in late February. The Committee reviewed the comments 
and completed its report 

In mid-March, the Analysis and Reports Committee's report was reviewed, 
modified and adopted by the Steering Committee. 

It is important to acknowledge that this report contains general 
recommendations and for any of them to be implemented, further refinement of detail 
will be needed. In some instances, there arc alternative ways of accomplishine the 
recommendation which must be explored and evaluated for technical, practical and 
economic feasibility. The development process, therefore, will need to continue. As 
NAEP continues the process of designing the form and content of the reports it will 
present to each state which participates m the 1990 mathematics assessment, the 
mterest of all parties and the very success of the process itself will be best served by 
continuing the invohrement of state education policjmakers, practitioners and scholars. 

GUIDING PRINriPLFi; 

During the process described above, a series of guiding principles emerged that 
miluenced the recommendations made in this report 

At its initial meeting in March, the Project's Steering Committee adopted a 
policy statement on the purpose of state comparisons and the conditions that should be 
met as follows: 

The purpose of State Level Smdent Achievement Comparison is to provide data 
on student performance to assist poliqmiakers and educators to work toward the 
improvement of education. Such data can be useful by encouraging and 



3 8 



detSSSS^ft^° ^ discussion of the quaUty of education and the conditions that 
State comparative achievement data can be useful if they: 

° leain"*^"^ performance on a consensus of what is important to 

0 Use sound testing and psychometric practices: 

0 Use procedures that minimize intrusion into instructional time; 

0 Take into account different circumstances and needs that the 
states face; and 

0 Are associated with features of the school systems that can be 
improved by policymakers and educators. 

of the s\eKl^^mmJni tJ^Analysis.and Reports Committee foUowed the position 
or me Meenng Committee, and durmg its discuss ons several major DrinciDles became 
mi^na for developing options and mSdng choices about reportig sKSlveS 



State comparative assessment can help citizens and state and local 
leaders identify and select educational improvement objectives if 
the data are sufficiently descriptive of what students know and do 
not know. 

•> 

Far more is needed than just an avera«e score of student 
performance. 

State-level trend data are important as state-by-state comparisons. 
ihcy provide encouragement to reform effom that seem to be 
matang a difference. They can indicate whether more effort is 
neeaea or "course corrections" are warranted. 

In addition to achievement data, assessments need to provide 
information on what educational variables seem to make a 
difference that might be affected by policies, programs and 
practice. 

Since comparative reports will be judgmental, they need to 
acknowledge that states differ in the economic and demographic 
settings m which schools operate and in the sodal and economic 
badcoround of the students entering the schools. These are 
vanablcs that over a short period of time may account for some 
part of the different outcomes among the states, and over a long 
penod of time, may determine the magnitude of the task ♦o 
achieve equal results. 
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II. ON REPORTING STUDENT ACHIEVEMENT DATA FOR EACH STATE 

One of the basic questions that data must answer is, "How w<;ll do students in 
each state perform on the assessment?" The most frequently used reporting rubric, 
particularly that used by the popular media, is to report the meari/or average numerical 
score of the population tested. An example is annual reportine of the average SAT or 
ACT scores of students in each state who took those tests. One o^:' the wavs NAEP has 
reported proficiency in a subject is by presenting averages on 3<itale of 0*- 500. To 
know that the average score of eighth grade students;ih a state is 270 on such a scale, 
provides, by itself very little information to the public, policy-makers or practitioners. 
With two or more assessments, trends can be shown by reporting averages, but such 
reporting provides no information about how the performance of students is distributed 
or what are. student strengths or weaknesses in various topics or areas of the subject 
assessed. Therefore, such assessments provide little guidance for those trying to decide 
where to focus instructional improvement efforts. 

REPORTING PROFICIENrV LKVKTi; 

Reporting the distribution of student scores in a state between quartiles is more 
informative than average scores and can reveal where changes occur if reported as 
trend data. Even more mieaninefiil, however, is the distribution of scores against a 
standard or set of standards. Of the various ways that state achievement data might be 
reported using a readily defined standard, NAEFs recent development and ajpphcation 
of proficiency scales seems to be an informative method that can also be easuy 
understood. 

Recommendation #1 

State achievement data will be reported bv scales similar to NAEFs proficiency 
scales. I.e.. the percent of the students in each grade who score in or above each of the 
defined proficiency levels and descrip tions of the performance of average smdents 
nationally at grades four, ei ght and twelve will be provided. 

The definitions for the five levels used in analyzing and reporting the results of 
the 1986 NAEP mathematics assessment are as follows: 

Level 150 - Simple Arithmetic Facts 

Learners at this J'^fsl know some basic addition and subtraction facts and can 
add two-digjt numbers without regrouping. They recognize simple situations in 
which addition and subtraction apply. They also are developing rudimentary 
classification skills. 

Level 200 - Beginning Skills and Understanding 

Learners at this level have considerable understanding of two-digit numbers. 
They can add two-digit numbers, but are still developmg an ability to regroup in 
subtraction. They know relations among coins, can read information firom charts 
and graphs, and use simple measurement instruments. They are developing 
some reasoning skills. 
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Level 250 - Basic Operations and Problem Solving 

^^^^ ^l?^ ^Zll ^ initial understanding of the four basic operations, 
c^!^ are able to add and subtract whole numbers and apply these sldSs to one- 
step word problems and money situations. In multipUcation, they can find tiie 
product of a two-digit and a one-digit number. They can also compare 

Level 300 - Moderately Complex Procedures and Reasoning 

Jfn^®" ^! ^iT^ developinK an understanding of number systems. Tbey 
can compute with decmials, simple fractions, and conSnonly encountered 
percents. They cfn identify geometric figures, measure lengths and ancles, and 
fn^Sv^*^ f rectangles. These students are also able to interpret simple 
mequahties, evaluate formulas and solve simple linear equations. They can find 
averages, make decisions on information drawn from graphs, and use loeical 
reasomng to solve problems. They are developing ±e skills to operate wth sien 
numbers, exponents, and square roots. "^ai^ii 

Level 350 - Multi-step Problem Solving and Algebra 

Learners at this level can apply a range of reasoning skills to solve multi-step 
problems. They can solve routine problems involving fractions and percent^ 
recognize properties of basic geometric figures, and work with exponents and 
square roots. They can solve a variety of two-step problems using variables 
Identify equivalent algebraic expressions, and solve linear equations and ' 
mequahties. They are developmg an understanding of functions and coordinate 

This is a dcvglopmcntal scale; tiie same scale is used by NAEP for reporting the 

fh?n?n.^Scl^°^?S^^^ ™s has th/advantage of ISSg 

he progress that students attam between grades. The scale was empincally derived- 
DrofiSrv"i.vSc T ?^°^5?.were identified tiiat were good dis?riminatoX^een 
Pfofiaencv levels. In "anchorins" tiie scale, items were selected whereby smdents at a 

Shfu ^^7 "^'^^ ^^l"^^^ 65-80 percem probabiUty of success witii tiiose questions 
while smdents at a lower level would be much less successful. 

What would also be desirable is some reference to a standard of what is 
cS?i,S^ ^ffu^^^ for students at each grade level to achieve. Of necessity, such a 

band) on the scale accompanied by descriptions of what 
Efc^SSSS objectives are appropnate to be learned at each grade. Setting standards 
has tijdmonalfy been a state and local function. It is doubtful that a conseSus could 
DC achieved among thestates on some grade level standards so tiiat tiiey could be 
apphed m reporting 1990 matiiematics achievement; consensus might not be possible at 

; ii.^x.appropnate, however, for NAEP to explore tiie feasibility of tiie states 
estabhshing such standards tiirough consensus. 

^,.^mu^ ^® meantime, a better understanding of what tiie average fourth, eightii and 
twem grader knows and can do m matiiematics would be well served by NA^ 
Iw o«f l^'^^^X description or list of tiie average smdents' proficiencies, descriptions 
tiiat are lar more detailed tiian tile scale descriptions presented above. Such 
descnptions would ^ve more meaning to an average grade-level number or band. Such 
toT ?^ whatjhe average smdem in the nation does know would also 

help each state and distiict dehberate tiie goals and standards it may want to establish. 

. a 



REPORTING BY CONTENT AREA 

A second basic question that reports should answer is, "What are student - 
strengths and weaknesses m the various areas of a subject?" NAEP reports have 
addressed the question in the oast The NAEP Reading Report Card reported trends 
from 1971 - 1984 in reading. Selected questions were categorized into tnree reading 
skill areas: literal comprehension, inferential comprehension and reference skills. 
Changes were reported in the average percent correct in each categoiy for each grade 
level between 1980 and 1984. The reports revealed which reading skills were improving 
and which were declining. This potentially useful information can be used by states to 
decide whether some changes in curriculum emphases are desirable. 

Recommendatinn #2 

State achievement data be reported hy vari ous mathematics ability and content 
areas to reveal the strengths and weakn esses of students in a state in each area. 

In a separate report for this project, a matrix for the 1990 math assessment is 
proposed. It contains three mathematical abilities: 

0 conceptual understanding 
0 procedural knowledge 
0 problem solving 

and five content areas: 

0 numbers and operations 
0 measurement 
0 geometry 

0 data analysis, statistics and probability 
0 algebra and functions 

The state reports on student achievement should include data on how well 
students in each §rade level tested perform in each of those ability and content areas. 
If the number of items is large enough, then achievement in each mathematical ability 
area within each content area should be reported. 

REPORTING TRENDS 

A third basic question that reports must answer is, "How does student 
achievement change over time?" In 1987, ei^t states contracted with NAEP to conduct 
a state math assessment using NAEP items. For those states which may also participate 
in a 1990 state level assessment, a trend report will be possible. In other states, trend 
reports will only be possible after two or more successive assessments. It will be m each 
state's interest to participate in successive assessments so that NAEP can be a source of 
information about whetner improvement is occurring. 
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, CrQS5-5CCtl0nai trend data, as thev are availahl ^ . be reported tn each state hv 
changes in the PrOPQrgQn of students SCOnn| r at e^^ pfofiri^n];^ iV v^l ^1 "^"g ' ^ 

ghang?s u i avcrsge achievement scores, m arirf.>m p : t^ARP^-iiV^nnrt T^^ for 
student achievement m each ahili|V af, " ^ ^n\ fint -j Si — ^^WtfaWT^r 

c ^ issue concerns the basis for trends. NAEP trend data has been cross- 
sectoonal, Lc., performance of 1986 ninth graders was compared to that of 1982 ninth 
StS: are based on longitudinal data. Students are assessed at one 

S S^Jr^ FSf ^5^^ ^ater. ITie Alexander-James 

studv report, Tbc Nation's Repi^rt Tflrd, recommended that this method be considered 
Whde the research potential in longitudinal studies is considerable, the burden on 
school systems to keep track of mdividual students over two to four years and the cost 
?nnSS^^*i"??^ * assessment to those students, wherever they are, seems excessive. 
Lonatudmal studies are valuable, but are best carried out through vehicles other than a 
assessment Furthermore, the focus of NAEPis on the performance 
of the educational wstem not mdividual students. How well the system has done with 
succewive groups of students can be answered with cross-sectional data. NAEP trend 
data should contmue to be cross-sectionaL 

Another issue is whether trend data should be reported only as chanee in 
average achievement or also on changes in percent correct in the content^as and by 

mprofiaency scale distnbution. Changes in the percentages of students in a 
state on different profiaency scales are far more informative than changes in an over- 
all average score. There is a difference in whether an increase in over^ average 
achievement is primarily the result of more students performing in an advanced 
category, or fewer scormg in a rudimentary category, or due to mcreases in both. At 
which level unprovements are occurring or not occurring has important impUcations for 
directmg where to focus improvement efforts. Whereas reporting trends in over-aU 
average achievement may ^ desirable to reveal the magnitude 6f a trend, the analytic 
advantage of reportmg proficiency scale trends is also considerable. 

Likewise, it can be very valuable to know what trends occur in student 
achievement m vanous mathematical process and content areas. Changes in over-aU 
average math scores do not reveal in which content or ability area the ciianee has 
occulted, information that would be useful in planning where to target improvement 



i» While data hke those descnbed above can be useful, it is important to 
f^^^l^l ^^J.f ® state sample of students assessed and the number of 

items mca<* ability and content area may limit the level of detaU that can be reliably 

exan^le, reporting assessment results by abiUty and topic could call for 
reportmg by a matrix of 5x3 or 15 cells. This is desirablJ, but NAEP need to 
cj^lore whether such a presentation for each state is feasible given the sample size and 
budget constramts to which it must adjust ai^«^«mu 

REPORTING R Y SUB^^ROTTP?? 

o^hu™? ♦T^ basic question that reports to the states should answer is "What are the 

^^p«"^^ s-'^^^^' type of 
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RernmmftTidatinn #4 

, Achievement data will be reported to Mt>h state for population sub-pnun. hv 
race/cthmcitv. gender, and tvne of communitv in whic h the stiident lives as long a-sThft 
numpgrs of students m each sub-group in the sample test e d are sufficient to providft 
stable and reliablfr estimates. : 

It can be helpful to policy-makers and practitioners to know whether 
miprovement is occurring m all types of communities: urban, suburban, and rural, or 
only in one or two types of communities. It can be important for leaders to know the 
magmtude of the levels of achievement of black, white or Hispanic students and to 
know whether improvement is occurring within each racial or ethnic group iii the 
student population. Monitoring the achievement of boys and girls in conjunction with 
enrollment in various courses can help state leaders decide whether there is a problem 
that needs attention. To begin to explore the possibility that problems involving the 
education of sub-groups of the student population requires that state achievement data 
be provided on various sub-groups. A note of caution: sub-groups of some pooulations 
are very small in some states. For example, Idaho had, as of 1984, a black student 
population of only 0.4%. Some states may not have enough urban students or enough 
rural students in the sample to provide valid representation. Whatever sample size m 
each state turns out to be economically feasible, it is likely that the number of students 
m some groups in a few states will not be large enough to provide valid results without 
over-sampling." Over-sampling should be a decision that a state makes jointly with 
NAEP. When the size of the sample of a sub-group is too small to provide stable and 
rehable estimates, then the report should not mclude achievement data on that sub- 
group. 

Traditionally, the national assessment has reported the race/ethnicity of 
students in three groups: black, Hispanic and white. Ihe number of other groups of 
students in the national sample has not been large enough to provide valid 
generalizations. In some individual states, however, there may be enough students or 
an interest in over-sampling to provide achievement data on native American Indians, 
Asian-Pacific stiidents or others. If requested by a state, NAEP should provide reports 
on groups other than black, Hispanic and white stiidents if the sample s^ is large 
enough. 

ILLUSTRATIVRT>VffT.y^<y 

In following the above four recommendations, the analysis and presentation of 
state data could take many forms: tables, bar charts, or line graphs. The forms should 
^^r5^ depending on the audience for the reports. Such decisions should be made by 
NAEP as it prepares its reports. The main data elements on which stiidents 
mathematics achievement would be reported for each state should include at least the 
following: 

0 Change in achievement over time 

0 Achievement of all students and of sub-groups identified by 
race/ethnicity, gender, and type of .commumty 

0 Achievement displayed as the percent of students scoring at or above 
each of the levels ot the proficiency scales 

0 Achievement data reported by the mean percent of selected items correct 
m each mathematics ability and content area 

9 ^ 



^* ^^^cs that illustrate one way of following the recommendations. The 
luuOwiiig iabics aavc cdluffiiis for 1992, as well as 1990 data to illustrate that the 
measurement of tiends is very important. 

In presaniii^ graphic displays of data, NAEP should not overlook the 
importance of maJaog narrative observations of what the analysis of data reveals. 
NAEP has alwsx-? aiade narrative observations in the past and should continue to do so 
for state-level assessments. 



thgmatlcs Proficiency For All Students And Bv Gender 
( State Grade 4 

All Students Female Male 

Scale '90 '92 Chg. '90 '92 Chg. '90 '92 Chg. 

150 

200 

250 

300 

350 

Average 



GRADES 

(The Same) 

GRADE 12 

(The Same) 



15 



10 



state Mathematics Proficieacv B y Race/Ethnicity 
i — tfvflw J ^rSQe ^ 

^ , glackgtuggnts Hispanic Students White Students 

Scale '90 '92 Chg. '90 '92 Chg. '90 '92 cig. 

150 

200 

250 

300 

350 

Average ^ 

GRADES 

(The Same) 

GRADE 12 

(The Same) 

State Mathematics Proficiency Bv Type of Commanity 

( Statg ) Grade 4 

Rvrsl Suburban Urban 

Scale '90 '92 Chg. '90 '92 Oig. Chg. 

150 

200 

250 

300 ; 

350 

Average 

GRADES 

(The Same) 

GRADE 12 

(The Same) 
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State Mean Sub-Scale Sgnr^ Bv Typ e Of Mflthpnifl|iy«f 
Oaestion Bv All Students AnH Ryjggmlgr 



( State ^ 
Ability . 



Conceptual 
Understanding 

Procedural 
Knowledge 

Problem 
Solving 

Combined 
(# Items) 



Grade 4 
All Students 
'90 '92 Chg. 



Female 
'90 '92 Chg. 



Male 

'90 '92 Chg. 



Ism 



Numbers and 
Operations 

Measurement 

Geometry 

Data Analysis 
Statistics 
& ProbabiUty 

Algebra & 
Functions 

Combined 
(# Items) 



All Students 
'90 '92 Chg. 



GRADES 

(The Same) 

GRADE 12 

(The Same) 



Female 
'90 '92 Chg. 



Mais. 

'90 '92 Chg. 



T7 
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state Mean Sqfr-Scale Score Rv Type Of M^^^ fi^mfl^f y.^ 



( State ) 
Ability 



Conceptual 
Understanding 

Procedural 
Knowledge 

Problem 
Solving 

Combined 
(# Items) 



Grade 4 
Black Studentq 
'90 '92 Chg. 



Hispanic ^itnd^^t^ White Students 
'90 '92 Chg. '90 '92 Chg. 



Ism 



Numbei-s and 
Operations 

Measurement 
Geometry 

Data Analysis 
Statistics 
& Prob£d)ility 

Algebra & 
Functions 

Combined 
(# Items) 



Black Students 
'90 '92 Chg. 



GRADES 

(The Same) 



Hispanic Student^ White Smdents 
'90 '92 Chg. '90 '92 Chg. 



GRADE 12 

(The Same) 



18 
13 



state Mean SulhScale Score Bv Type pf Ma»hpm.,ti^^ 
Onestion Bv Type Of Communis 



( State ) 

A bilit y 



Conceptual 
Understanding 



Grade 4 

Rural 

'90 '92 Chg. 



Suburban 
'90 '92 Chg. 



'90 '92 Chg. 



Procedural 
Knowledge 

Problem 
Solving 

Combmed 
(# Items) 



Numbers and 
Operations 

Measurement 

Geometry 

Data Analysis 
Statistics 
& Probability 

Algebra & 
Functions 

Combined 
.(# Items) 



'90 '92 Chg. 



Suburban 
'90 '92 Chg. 



mm 

'90 '92 Chg. 



GRADES 



(The Same) 



GRADE 12 



(The Same) 



18 
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If feasible, a more desirable alternative to the forgoing tables would be as 
follows: 

(Statg) Grade 4 

All Students Female Male 

'90 '92 Chg '90 '92 Chg '90 '92 Chg 

Numbers and 
Operations 
Conceptual 

Understanding 

Procedural 

Knowledge ' 

Problem 

Solving 

Measurement 
Conceptual 

Understanding 

Procedural 

Knowledge 

Problem 

Solving 

Geometry 
Conceptual 

Understanding 

Procedural 

Knowledge 

Problem 

Solving 

Data Analysis 
Staiisilcs and 
Probability 
Conceptual 

Understanding 

Procedural 

Knowledge 

Problem 

Solving 

Conceptual 

Understanding 

Procedural 

Knowledge 

Problem 

Solving 



20 
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III. ON REPORTING STATE-BY-STATE COMPARISONS 



^ In addition to providing useful infonnation about the proficiency of students in 
eacu staic, asscssffieais need to report how weil students in one state perfonn in 
comparison to the jperformance of students in other states. If the data are detailed 
enough about levels of proficiency and about each mathematical ability and content 
area, then comparisons can serve a purpose of helping state leaders to select and target 
desu-ed improvements. In addiuon, the comparisons need to acknowledge that states 
differ m tho education needs of the students they serve. During the deUberations of the 
worlong committee and m responses firom the field to proposals in a draft, interest in 
and support for three kinds of comparison reports emerged. One would take into 
account the differences in those demographics in each state that are likely to relate to 
educatmg students such as per capita income, educational level of the adult population 
and percent of students m poverty. A second would take into account the background 
vanar.es of the students who are assessed. Both of the above would result in 
assessment reports which acknowledge that states differ in the nature and magnitude of 
their educational task by characteristics of the general population they serve and the 
children they teach. In addition, considerable mterest was expressed m receiving 
reports which make it easy for Iea;ders in a state to examine its students' achievement in 
comparison to that of students in nearby states. 

Three forms of state-comparative reports are, therefore, proposed. 
RANKING BY STATE DEMOGRAPHIC VARIART.F.< ; 
Recommend ation #5A 

The achievement of students in ftarh ^itatft wij] \ ^ displayed in a ranking of the 
St ates based on a composite of state demographic variables. 



The main task is to choose the most appropriate demographic indicators for 
ranlang the states. An in-depth examination of various alternatives was not possible 

techmcal experts, and in consultation with state representatives, should select a basis for 
demographic ranking. 

In another project of the Council of Chief State School Officers, a series of 
rehable and readily available set of education related indicators has been identified and 
presented m a report, Volume I: State Education In dicators 1987. Three of the 
mdicators seem to give the most appropriate reflection of differences in the states that 
relate to the background of the students in tiie schools and are used here for the 
purpose of illustration. For each state, these are: per capita income, the percent of 
adults who have completed four yars of high school ana the percent of school age 
chUdren who live in poverty. The per capita income indicator is 1S36 data from the 
U.S. Bureau of Economic Analysis. Information on education attainment level and 

population in poverty i^ from the U.S. Bureau of the Census for 
1980. Hopemllv, by the time state comparative reports for a 1990 mathematics NAEP 
are issued, 1990 census data will be available. From one perspective, the 1980 census 
data are appropriate for use. Many of the children who will be assessed in 1990 were 
bom m 1980. They, as well as those bom earlier, have lived a substantial period of their 
uves durmg which 1980 census data closely approximated the demographics of their 
environment In aity event, other measures of income, educational attainment and 
school-age poverty might be considered. For example, state income per pupil might be 
more desirable than per capita. Education attainment of adults age 25-55 might be 
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better than that of aU adults. In addition, factors such as percent of high income 
famihes ^d percent of college graduates might be mcludk The spedfic types and 
sources of data need to be further explored. ^^-vyy^^xnm 

fh.nuah^uJ?^l? d?°io«raphic ranldng, presented as an iUustration, was created 
s^^ A^e SeSl? '^^^ "^'^^ °° 

cto«^,,7!f ^5?^"^?° 0^ each indcator was standardized to a mean of zero and a 
standard deviation of one: Then, the composite index was computed for each state 
gvmg equal weightmg to each of the three components and addSae their values 

^""^^ ^ ^ ^^"^^P*' sign was reversed States were then 

ranked by the composite mdex. 

The composite and the standardized scores for each of the three variables is 
second table. The third table represents one way the achievement data 
could be presented. TTie states are hsted by composite rank, the base data on which the 
composite was denved are displayed and the achievementdata presented: 

onmxlS^L^^^^^^A®^ ^ °^ presentation is that any state can create its own 
chSaStira^ identifymg and clustermg a group of states with sinular demographic 
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STATE DEMOGRAPHIC VARIABLES 





1986 


1980 Percent 


1980 




Per-Capita 


Adults 


Percent School Age 


STATE 


Income 


4 years H.S. 


Population in Poverty 


Alabama 


11,336 


56.5 


22.1 


Alaska 


17,796 


82.5 


11.0 


Arizona 


13,474 


72.4 


15.4 


Arkansas 


11,073 


55.5 


22.3 


California 


16,904 


73.5 


13.8 


Colorado 


15,234 


78.6 


10.5 


Connecticut 


19,600 


70.3 


10.2 


Delaware 


15,010 


68.6 


14.4 


Florida 


14,646 


66.7 


17.2 


Georgia 


13,446 


56.4 


20.1 


Hawaii 


14,886 


73.8 


11.4 


Idaho 


11,223 


73.7 


13.1 


Illinois 


15,586 


66.5 


13.9 


Indiana 


13,136 


66.4 


10.8 


Iowa 


13,348 


71.5 


10.6 


Kansas 


14,650 


73.3 


10.5 


Kentucky 


11,238 


53.1 


20.7 


Louisiana 


11,193 


57.7 


22.6 


Maine 


12,790 


68.7 


14.8 


Maryland 


16,864 


67.4 


11.6 


Massachusetts 


17,772 


72.2 


12.1 


Michigan 


14,775 


68.0 


12.2 


Minnesota 


14,994 


73.1 


93 


Mississippi 


9,716 


54.8 


29.8 


Missouri 


13,789 


63.5 


13.7 


Montana 


11,803 


74.4 


12.5 


Nebraska 


13,742 


73.4 


11.4 


Nevada 


15,437 


75.5 


9.0 


New Hampshire 


15,911 


72.3 


8.7 


New Jersey 


18,626 


67.4 


13.2 


New Mexico 


11,422 


68.9 


21.2 


New York 


17,111 


66.3 


17.5 


North Carolina 


12,438 


54.8 


17.5 


North Dakota 


12,472 


66.4 


13.7 


Ohio 


13,933 


67.0 


12.0 


Oklahoma 


12,283 


66.0 


14.7 


Oregon 


13,328 


75.6 


10.4 


Pennsylvania 


14,249 


64.7 


13.0 


Rhode Island 


14,579 


61.1 


12.4 


South Carolina 


11,299 


54.0 


20.3 


South Dakota 


11,814 


67.9 


19.0 


Tennessee 


12,002 


56.2 


19.8 


Texas 


13,478 


62.6 


18.1 


Utah 


10,981 


80.0 


9.6 


Vermont 


13,348 


71.0 


12.7 


Virginia 


15,408 


62.4 


14.1 


Washin^on 


15,009 


77.6 


10.0 


West Virginia 


10,576 


56.0 


17.9 


Wisconsin 


13,909 


69.6 


9.5 


Wyoming 


12,781 


77.9 


7.4 
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STATES RANKED ON COMPOSITE OF STANDARDIZED DEMOGRAPHIC 

VARIABLES 

PercenU Per Capita Percent 

Above Pov. 



STATE Composite Yrs.H.S. Income 

Neval ifAl 1-6094 0.6094 

Nevada 3.6135 1.1177 0.8722 16235 

Connecticut 3.5502 0.2779 2.2433 1 0290 

NewHampsbre 3.3697 0.4932 1.0290 I8475 

Wyommg 3.2658 1.4600 -0.4376 2 2433 

Q)Iorado 3.1948 1.6235 0.7344 0 8368 

Washmgton 3.0554 1.3283 0.6094 1 1177 

Mmnespta 2.6197 0.6094 0.5503 l'4600 

Massachusetts 2.2278 0.4376 1.4600 0 3301 

Hawau 1.9626 0.9477 0.4932 0 5218 

Kansas 1.8907 0.6706 0.3833 0.8368 

Oregon 1.8861 1.2163 -0.2779 0 9477 

Calilorma 1.8422 0.8014 1.2163 -01755 

New Jersey l.722() ^0.1000 1.8475 -00249 

Wisconsin 1.6797 0.2264 0.1250 1 3283 

Maiyland 1.4553 -0.1000 1.1177 04376 

^tah , 1.4402 1.8475 -16235 1 2163 

Nebraska 1.2811 0.7344 0.0249 0 5218 

K^?.. 0.9168 0.3833 -0.2009 0 7344 

Miciiigan 0.7404 0.0249 0.4376 0 2779 

Illinois 0.4434 -0.2779 0.9477 -02264 

Delaware 0.4153 0.0749 0.6706 -03301 

Montana 0.4031 1.0290 -0.8014 o'l755 

0.3833 -0.1755 01755 0 3833 

Vermont 0.2542 0.3301 -0.2009 01250 

New York 0.2508 ^0.4376 1.3283 -06400 

Indiana -0.0163 -0.3567 -0.330! 067C5 

Arizona -0.0177 0.5503 -0.0749 -04932 

Virgmia -0.2109 -0.7344 0.8014 -02779 

Pennsylvama -0.2490 -0.5503 0.2264 0 0749 

Rliode Island -0.2971 -0.8014 0.2779 0 2264 

Idaho ^.3191 0.8722 -1.2163 0 0249 

Flpnda .OM66 -0.2264 0.3301 -05503 

Missoun -0.6344 -0.6094 0.0749 -o'lOOO 

Mame -0.6959 0.1250 -0.3833 -04376 

North Dakota -0.9499 -0.3567 -0.4932 -o'lOOO 

Oklahoma -1.2858 -0.4932 -0.6094 -0.3833 

Texas -1.4970 -0.6706 -0.0249 -08014 

SouthDakota -1.6316 -0.0249 -0.7344 .08722 

New Mexico -2.0250 0.1755 -0.8722 .l'3283 

S^^^iP^ -2-1830 -1.0290 -0.1250 .10290 

North Carolma -2.7321 -1.5418 -0.5503 -0.6400 

Tennessee -2.7360 -1.1177 -0.6706 -09744 

Louisiana -3.2824 -0.8722 -1.3282 -16235 

Alabama -3.7429 -0.9477 -0.9477 -i 8475 

WestVirgima -3.7982 -1.2163 -1.8475 -07344 

South Carohna -3.9942 -1.8475 -10290 -11177 



Er|c 19 ?4 



(Cont'd) 

QTATn n Per Capita Percent 

STATE Composite Yrs.H.S. Income Above Pov. 

Arkaiisas -4.2484 -1.3283 -1.4600 -1.4600 

Kentucky ^ -4.5773 -2.2433 1.1177 -1.2163 

Mississippi -6.0284 -1.5418 -2.2433 -22433 



Based on illustration. 

Prepared by Program Evaluation and Research Division, 
California State Department of Education. 
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STATES RANKED ON COMPOSITE INDEX OF DEMOGRAPHIC VARIABLES 



1986 1980% 1980% 

Per Adults School 

Capita 4 yrs Age Pop. 

Income H.S. in Poverty 



Achievement Grade 4 
Proficiency Scales 



STATES 








Alaska 


17796 




11 n 


Nevada 


15 437 


7SS 


0 n 


Connecticut 






xU.Z 


New Hampshire 


1S911 

Xa/|TXX 


773 


o./ 


Wyoming 


12.781 
#ox 


770 


74 


Colorado 




78 6 
/O.O 


in < 


Washinizton 


xj|Uv/y 


776 


in 0 


Minnesota 




TX 1 




Massachusetts 


17 772 


777 


17 1 


Hawaii 


l4ftR/^ 

X^|O0U 


73 8 


11 4 
11.*+ 


Kansas 


14 650 


73 3 


10 S 


Oregon 




7S6 


in 4 


Califoriia 


16,904 


73^ 


13 8 


New Jer^y 


18,626 




137 


Wisconsin 


13 909 


60 6 


0 s 


Maryland 


16 864 


674 


11 6 
xx.u 


Utah 


10,981 


800 


06 


Nebraska 


13 742 


734 


11 4 

XX.H 


Iowa 


13 348 


71 S 


in 6 


Michigan 


14 775 


680 


17.7 


Illinois 


15.586 


66^ 


13 0 


Delaware 


15,010 


686 


144 


Montana 


11,803 


744 


12^ 


Ohio 


13 033 


67 0 

U 1.1/ 


17 n 

lAU 


Vermont 


13.348 


71 0 


17 7 


New York 


17111 

X f ,xxx 


66 


17S 


Indiana 


13 136 


664 


infi 


Arizona 


13,474 


7Z4 


154 

Xk/.*T 


Virginia 


15,408 


6Z4 


14.1 


Pennsylvania 


14^9 


64.7 


13.0 


Rhode Island 


14^ 


61.1 


1Z4 


Idaho 




73.7 


13.1 


Florida 


14,646 


66.7 


17.2 


Nfissouri 


13,789 


635 


13.7 


Maine 


12,790 


68.7 


14.8 


North Dakota 


12,472 


66.4 


13.7 


Oklahoma 


12,283 


66.0 


14.7 


Texas 


13,478 


62.6 


18.1 


South Dakota 


11,814 


67.9 


19.0 


New Mejdco 


11,422 


68.9 


212 


Georgia 


13,446 


56.4 


20.1 


North Carolina 


12,438 


54.8 


17.5 



150 200 250 300 350 Aver. 
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(Cont'd) 



1986 1980% 1980% 





Per 


Adults 


School 




Capita 


4 yrs Age Pop 




Income 


H.S. in Povert 


STATES 








Tennessee 


12,002 


562 


19.8 


Louisiana 


11,193 


57.7 


22.6 


West Virgijiia 


10476 


56.0 


17.9 


Alabama 


1U36 


56.5 


22.7 


South Carolina 


11,299 


54.0 


203 


Arkansas 


11,073 


553 


223 


Kentucky 


11,238 


53.1 


20.7 


Mississippi 


9,716 


54.8 


, 29.8 



Achievement Grade 4 
Proficiency Scales 

150 200 250 300 350 Aver. 



National Average 
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COMPARING THE ACHIEVEMENT OF SIMILAR STUDENTS 



Whereas the above method of comparison is based on differences in the 
demographic characteristics of each staters general population, another way is to base 
the comparisons on the social and economic differences in the background of the 
students who are being assessed. States differ considerably in the percentage of 
students tested who are advantaged and disadvantaged in terms of their pre-school and 
out-of-school experiences. All states have substantial numbers of advantaged and 
disadvantaged students. It is, however, the differences in the percentages of such 
students in each state that make the nature and magnitude of ie state's educational 
task different. This leads to the second of the two recommendations on the 
presentation of state comparisons. 

Recommendation #5B 

State compari sons will be presented on the achievement of groups of students 
tested who have similar backgrounds so that similar students are compared. 

ff students can be grouped reliably by similar background characteristics, 
comparisons of similar students in different states can be made, even though the 
number or percent of students in a particular grouping will vary from state-to-state. 

Obtaining reliable student data for establishing groupings is the challenge. It 
would seem desurable, for example, to establish at least three comparison groups such 
as disadvantaged, advantaged and hi^y advantaged. Historically, NAEP has asked 
students.a senes of background questions that reasonably provide for the creation of 
groupings and the classification of students who are assessed. These questions relate to 
parent education, numbers of books and periodicals in the home, and other pertinent 
factors. The answers about parent education from students in the fourth grade are not 
as reliable as those received from eighth and twelfth graders. The objective, however, is 
to create reliably consistent clusters of students and not to determine the precisely 
accurate assignment of individual students. NAEP would need to construct and 
validate the use of student questions for this purpose. 

One alternative is to secure student social and economic background data by 
asking the students' parents. A telephone survey of the parents of students who are 
assessed may be a feasible procedure. A telephone survey of a sample of parents is 
used successfully in a number of states and local districts. It could provide a wealth of 
educationally relevant opinions and facts as well as information about parent education 
level, income range, and other SES data, 

A second alternative way of creating groupings of students and assigning 
individuals that may be feasible is to use data from me United States Census that are 
aggregated by zip <x>de or postal carrier routes. Zip codes or carrier routes would be 
used to establish a grouping of disadvantaged, advantaged and highly advantaged based 
on such census data as average parent income and education level as well as 
occupation. While the data are not about the specific parents of the individual students 
assessed, they may be as reliable as student selt-reports. NAEP should explore the 
feasibility of creating groupings of similar students with census data by zip code or 
postal route. 

These and other alternatives described above should be evaluated by NAEP and 
the.most feasible one selected. 
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One way to present state comparison data on similar students would be as 
follows: 

MATHEMATICS PROFIClENrV OF DTSA W A^ TTAGED STUDENTS 

Grade 4 

Scale 

Percent of 

S^pl^T^$ted m m 2^ m m Mean 

Alabama 

Alaska — — — 

Arizona 

Arkansas — 

California — — 

Colorado — 

Connecticut — 

Delaware — — — 

Florida ^^^^ 

Georgia — 

Hawaii — — 

Idaho ^^^^ — — 

Illinois — — 

Indiana — 

Iowa — — — 

Kansas — — 

Kentacb^ 

Louisiana — — 

Maine 

Maryland — — 

Massachusetts — 

Michigan — — 

Minnesota 

Mississippi — 

Missoun ; 

Montana ' — 

Nebraska — — 

Nevada — — — 

New Hampshire — — 

New Jersey — 

New Mexico ' 

New York ~ — — 

North Carolina 

North Dakota — — 

Ohio — — — — — 

Oklahoma — — — — 

Oregon — — — 

Pennqrlvania — — — 

Rhode Island 

South Carolina — — 

South Dakota — — 

Tennessee — — — 

Texas — — — 

Utah — — — — — 



ERLC 



^9 

24 



(Cont'd) 



Percent of 

Sample Te?itgd m m m m m Msaa 



Vermont 

Virginia 

Washin^on 

West Virginia 

Wisconsin 

Wyoming 



The same presentation would be made of data for advantaged and hiehlv 
advantaged stiiffy^tf; in aU three grades tested. ^^^gcu <mu 

In addition, achievement data would be presented for each group of similar 
smdents m each grade level by each mathemati^ abiUty and conteS are^ 

COMPARING ACHTIi:VF,MF1SJ T IN NRAR.RV f STATlTg 

There is a desire expressed by a number of state leaders to be able to 
conveniently compare the achievement of smdents in their states with that of smdents 

°;5.T^yu'^'^- ^^"^^ respondents to an earUer draft of th^ report 
recommended that such a presentation be provided. A recent report from the Council 
nl^nt^H ~Sn?P^^ ^^^"^ entitled Volume T: 5;t.t^. F^„r.Wn^ Tn^.-^^»^ 1007 
fnS1?c?c^,T ^oupmgs as a^efulwayot c^phyiig^;^^ 
S?N^S5n?^ achievement, Ilie same re^onafgroupings were used in a report for 
the National Governors' Association. Time I^r R<»^its'> l7 ^ 

Recomme ndation 

Tbg achigvCT^nt of students will be presented in rft g ional clusters nf 'it^ t?., 

ornnniJJS f^"^ fi^u ^^h statc leaders whetiier tiiese particular 

SnSS^i • ""^t^^^ ^ comparisons, tiiey do seem reasonable to 

Jose involved m prep^g this report. Achievement data would be presented by 

?Sff 2f performance leveL in matiiematics abiUty ahd topic areas by 

regional groupmgs bke the following: f v <uc«> 
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REGIONAL CLUSTERS 



SOUTH ATLANTIC 

Florida 
Geor^a 
North Carolina 
South Carolina 
Virgmia 
West Virginia 

WESTSOLTH 
CENTRAL 

Arkansas 
Louisiana 
Oklahoma 
Texas 

MOUNTAIN 

Arizona 
Colorado 
Idaho 
Montana 
Nevada 
New Mexico 
Utah 
Wyoming 

PACIFIC 

Alaska 
California 
Hawaii 
Oregon 
Washington 

NEW ENGLAND 

Connecticut 
Maine 
Massachusetts 
New Hampshire 
Rhode Island 
Vermont 



COMPARING ACHIEVEMENT 



MID-ATLANTIC 

Delaware 
Maryland 
New Jersey 
New York 
Pennsylvania 

MIDWEST 

Illinois 
Indiana 
Michigan 
Minnesota 
Ohio 
Wisconsin 



WEST NORTH 
CENTRAL 

Iowa 
Kansas 
Missouri 
Nebraska 
North Dakota 
South Dakota 

EAST SOUTH 
CENTRAL 

Alabama 
Kentuclqf 
Mississippi 
Tennessee 



Source: National Governors' Association. Time for Results: 1987 
Washington, D,C: National Governors* Association 
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IV. ON RELATING ACHIEVEMENT TO EDUCATION VARIABLES 

The purpose of a state-level assessment should not be to just measure 
achievement, it should also be one source of information about what education 
variables seem to make a difference in the achievement of students, variables that can 
be influenced by jwlideSj programs and administi-ative practices. The value of state 
assessment data m the improvement of education will be gireatly enhanced if 
information about what seems to make a difference is secured, as well as how well 
students perform. 

The major question is, "Which educational context data should be collected and 
analyzed? It should be limited by how much is reasonable to ask of teachers, principals 
and others. It must be limited by a concern for the quality of the data. If the only 
feasible way to secure particular infonriation results in data that are weak, then the data 
should not be collected. NAEP cannot answer all research questions. Some can be 
better and more economically answered through other kinds of research and assessment 
projects. Nevertheless, NAEP can and should be one source of information on what 
seems to make a difference so that state leaders, in examining NAEP results and 
information from other sources, can make reasonable decisions about how education 
might be improved. 

During the process of considering what education experience data should be 
collected and examined, many variables were considered. One consensus that emerged 
Avas that the topics should be limited not only for the reasons stated above, but also 
limited to those that might be influenced, at least in mamr states, by state policy or 
programs or practices, m addition, a consensus emerged on the view that the first 
attempt to collect education experience data at the state level should be a cautious one. 

Recommendation #n 

Data will be seaired. analyze d and reported on relationships between 
achievement and educat ional variables that can he affected by educational policies. 
programs or practices. Examples of such variables are: 

0 The number of mathematics courses the student has completed in high 
school (Grade 12) 

0 The types of mathematics courses the student has completed or the 
advanced.level courses the student has completed such as Algebra 2, 
Geometiy, Statistics. (Grade 12) 

0 The amount of time the student spends on mathematics homework. 
(Grades 4, 8 & 12) 

0 The actual time (minutes per week) that students receive mathematics 
instruction. (Grades 4, 8 & 12) 

0 Whether students have had the opportunity to learn through class 
instruction specific topics within a process or content area of 
mathematics. Have certain topics been part of the scheduled and 
' presented instructional program? (Graces 4, 8 & 12) 
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0 Whether teachers of mathematics have a state certificate to teach 
mathematics. (Grades 4, 8 & 12) 

0 Years of experience teaching mathematics. (Grades 4, 8 & 12) 

0 Number of courses or credits teachers of mathematics classes received in 
college and graduate school in mathematics and in mathematics 
pedagogy. (Grades 4 & 8) 

0 Whether teachers of mathematics have college maiors or minors in 
mathematics. (Grades 4, 8 & 12) 

0 The extent of the mathematics teachers' recent professional development 
m mathematics or mathematics education (graduate courses, inservice 
workshops, conferences, meetings, etc.). (Grades 4, 8 & 12) 

0 Per pupil expenditures. 

The specific method of coUectine data on the above variables is beyond the 
purview of th^ particular report. NAEP has access to technical experts on its staff and 
eisewnere to design the appropriate questions or methods to secure useful data. 
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V. ON SELECTED PROCEDURES 
THE INCLUSION AND EXCLUSION OF SPF/^ IAL RDUrATTON STUDENTS 

,„o%^' ^^^^^ decision to exclude from testing a special education student 
who was drawn-^in the sample has been made by local school staff members 
SnK? f '^J'^^^S'^Ji^l' . '^^ coordinator was guided by definitions and 
procedures provided by NAEP and iiidividual reports that explained the basis for tiie 
exclusion were Drovided to NAEP for each child Secluded. In the past, this process has 
worked relatvely well for the national assessment With state comparis^ Sn Se 
honzon, the process warrants a review. Scores are very sensitive to exclusion or 
mclusion of students with limitations. lu "^Awuaion or 

To examine the issue of inclusion or exclusion of special education students, a 
reoort was Provided by tiie N itional Association of State birectors of Special 
bducation (NASDE). It contains many constructive recommendations that go beyond 
gie scope of tius report and so is being transmitted td NAEP and tiie Department of 
Education as a supplemental resource paper. Copies are available from NASDE. It 
was a prmciple source of ideas tiiat led to tiie foUowing recommendations: 

Recommendarinn #7 

0 All Students and all schools will be considered as potentially ftlig |>T^. f^j 
g ^lgn m XW^ NA F F spmPlV to 17^ tested Im^aT^^^ 
m SOlglv for -nal students and do not remn v e a special schnnl frnn^ 
ty ilSt unless there is a determination at the, s chool that Tinn<» fhff 

lists from which the g^ir rple will h& draw^ ^. 
0 NAEP Mil l continue the practice that the ria d sion to inrlnd^ or exclude a 

spefflsl student be made at the local school levfti. hiyi specify that the 
degsioq be made on a team basis consisting o f tiie p arentis ), teacherrs^ 

and Qtheg worinng with the student n.e.:thVrRP t^^mS l ^ie shidem's 

current ffiP contains recommendations reg arding group j^^^ndarHiV^rf 
testing, then they should be annlied making a s peaal meeting 
unnecessary. 

0 Newcriteria wiU be p rovided to KA^ f;p ^ [^^ ^-r^^ rdinators and TF.P tPam^ 
regarding inclusion /eyclnsm n as follows; 

when in doubt include the st"d?P» 

include all students placed in general educ ation setting s unless thf> 
lEP team spefflfif^lj y decides to exclude them 

jo not exclude anv students only on the basis of categ ory of 
handigg p 
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gXCtujg SftidentS where exoerienre and propam plans rfn nnt 
mclude ami nfivcr have included fh^ fP.t rn' Atent ^.g ,. whVn 
focus has been on life skills r^^ ther than nn ^ ^ft^ff^ 

e?<clude students who cannot rfispnnrf tff tf^T i tems even with 
accommori;^tlffns. 

The USg of the excluded student g uestinr^ngi ry unii rn^^irj^^ ^, and NAF.P 
wUl report for each state. th^nnn^h^rTnd P ^ ^^^^ ^^^tiHr"T? f" \4 

gOP^n who weV; e^chVd^ri hv r^f^y,^; inn^nT^^yf^ 
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0 COfffidemial codes will be indnded on thft fPQ t forms that id^ ntifa ' 
SpdCTg who tyfi tfte test and do have handfcap s /as ; iiS^hv an 
Ife-P). Report for each state the number and p ercent (ygp f.Hali;^T& 
who were tested, hv tvp^ nf 

0 NAEP should seek the resources that would he required an^ d?-^?'^p 

largg pnnt. braille, signed direction, mechaniri^i f ijdes.Ttr,7n tfi ^t^ 

students who, wnb accommodations, can talri» th^'t ^l^'^'^e j^^^^^^ in th^. 
assessmerjt, 

THE gXCLT^SION OF rJMTTED ENfiT JSH P R OFirTRNT ST^ myMT^g 

. i;^v ^ ^« national assessment, the decision of whether to exclude a student who has 
a hmited.proficiency m Endish has been handled by the same procedure med to makf 
the decision about a special education student. (See Appendix E for a summary of the 
S?S^"i?' loca^.scfiool coordinator, with the adviceXther staff meS^ay 
t^^d L^S^wrf"^ speakmg students - those who do not read or speak EngUsh Ld 
would be unable to overcome the language barrier in the test situation." 

provide^to NAEP $S ^ "^"^^'"^ ^'^'^^ 

While appropriate for national achievement data, the criteria for exclusion is far 
less appropnate when stete comparisons are to be madt TheTcSemem resiS V 
IZln^T ^ ^- ^«^<^!^y affected because they have a W^Se? o 'new 
students from foreign cpuntnes for whom English is a second iage, but who would 

exer^'^T^ 1? ^ft '2"^' Proficiency and can r^plnd to some NAEP 
exercises. They aire hmited m English, but not incapacitated. 

,n.thAiS*^S!!°^ the ejects of limited English proficiency (LEP) from 
SfnnS.^fS?'^ consktent mamier Is very di^idt Securing consistent 

apph<»tion of some .-ntenon across the United States is even more difficult j^T 
fS^^^It states have laws that prohibit the use of tests in English with LEP 
sSdSoilK consistent criterion would be the Sunt of time a 

sSes SLT&Fjtt'L^^^^ ^P«^t ^ a school in the United 

States. Most states and distncts expect that students who enter school in the U S and 

Hm?^aSon^ ^'J'^ Pr?ficiency in English will have overcome STmost of «ieir 
TS^SS^nrTcH °r It ^common for states to aUow the exclusion of an 

LE^ foreign-bora student from a state testmg program for a limited period of time. On 
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the other hand, some students from other countries overcome the English limitation 
more rapidly than others. The following is therefore recommended: 

Recommendation #8 

NAEP criteria for the exclusion of non-English .^peaking students will he 
replaced bv a guideline for the eychisin iLQf Limited English Proficient students a.s 
Mq^', Apv StUdgn t who entered scho o l i n the U.S. with limited English proficiency as 
qg grmmed by the school will be excluded from the naep t est if the smdent ha.s heftn 
in U.S. schook for less than three full sch ool years, unless in the judgement of the 
mwl Staff, the student's English proficiencv is equal to t h at of his or her proficiency in 
the native language. All students with t hree or more fiiU school vears in U.S. schools 
are to be tested. 



WHO SHOU LD ADMINISTER THE TESTS TO STUDENTS? 

In the past, emj)loyees of a NAEP sub-contractor have administered the tests to 
students. This minimized the data collection burden on local school staff. That 
oractice is partly the basis for widespread confidence in the conditions under which 
NAEP data have been collected, but it is an expensive part of the process. In 1986 and 
1987, a number of states carried out state assessments that permitted comparisons to 
the nationd NAEP. School district employees (not teachers) administered the tests. 
Opinions differ, but there is no clear evidence that one method is more or less reliable 
than the other or whether differences in the methods bring different results. 

Research is needed on the possible effects of different procedures. Only the 
results of such a study can reasonably address this issue. It would be tragic to mscover 
after-the-fact that a change to local staff administration adversely affects the results of 
state-to-state or state-to-nation comparisons. Until tiiat question is answered, the 
credibility of state-to-state comparison will be at risk if past practice is changed. 

Recommen dation #9 

The administrarion of state-le vel assessments in 1990 will be carried out by 
emPlqvees of NAEP or its contractor an d federal iiinds should be budgeted accordingly. 
A proje ct SDQUld be conducted bv NAEP whereby a small hut sufficient number of tests 
arc administered bv local district employees to determine whether who a riministerv thf^. 
tests makes a difference. In addition. NAEP. in cooperation with interested states, shall 
develop procedures whereby local district employees can, in a reliable wav. administer 
the tests beme used for state comparisons or being u sed to collect information at the 
school Of district leyel. 

REPORTING THE ACHIEVEMENT OF PRIVATE SCHOOL STUDENTS 

The national assessment has always been an assessment of all students in the 
United States, private school as well as public. It seems reasonable to continue, that 
practice. 

In most states, the percent of private school students is too small to provide a 
rehable measure without over-samphng, so the comparison of public and private school 
students within a state is not feasible in any but a few states. 
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nor* .,^°^"«?P<>°sible^or the effectiveness Of edu^^^ For the most 

part, It IS those elected and appointed government officials and educators who are 

StPeTri^Jnivfn^" r°^' ? '^^^"V states!SoffiSh^5e 

Recommendflti>n #10 

^AEP will continue to include privatp. w ell a.^ n nl^liV school students in its 

achievement will he m- pnh uc school stin jgpts ft"ly. • 

VI. CONCLUSION/FOLLOW-UP 

level as2t^enHn'?<j5nTn^/^^ be done to adequately plan for conducting state- 
rtromS^Snn? ^ ^«P?^« ? « results of that assessment Some of the 

f^S^r.n^l5*'°°^ 'IP^" ^ implemented in a variety of ways. To 
implement the recommendations m the other major report of this oroiect that Hefines 

^^^'^^^'^''^^^^^^^^ ^^/^ 1990 aiessTnt 2 t^eSe^i?em^'^^^ 
S tSrS.^?"^ exercises need to be written and reviewed. These observations lead 
to the finai recommendation in this report vauum ic<iu 

Recommendation #11 

iqqn ...^ the dgvelonment of the a■sse■s■sn^ ff■ pt instrument tn be used in the 

™ "^^^^ a.msment. the wav m which the ass;; iment wIll a^ ^ iyistlred and thl 

continug thg co nsensus nror^ - ss '^im edhythi^j ^r^^ ^^^ ^" 
poUcvmaKgrs . pr?^(lit gen> and Solars in iS S'^^rSn^ ^^ ^ 

Recommendatinp f ^2 




orocesZlSaSf m^?;-^'*^^!^'"?,^ achievement is a complex 

^L^Jt ZTJS^^^^^ ""^^^ ^ addressed and resolved before such 

^ttSS!!? ^® conducted, far more issues than this report addresses. The resolution 

WA^Jg^^^^f.l^^^tio'i pSblic policy leaded aaSe 
Umted States. State level NAEFs will be voluntaiy and the decision to Darticioate will 
be made by the appropriate decision-makers in.each state, one^te at aS The^ 

S?r,?^^P^^ ^ ^ ^ true. howIveXtXfis a^onJ^^^^^ 

amon^ the hundreds of people who have participated, and are wSto coSe to 
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APPENDIX A 

NATIONAL ASSESSMENT PLANNING PROJECT 
STEERING COMMnTEE 

American Association of School Administrators 

James E. Morrell - Superintendent of Public Schools, 
Muhlenberg, Pennsylvania 

American Federation of Teachers 

Antonia Cortese - First-Vice-President, New York State 
United Teachers, New York 

Association of State Assessment Programs 

Thomas Fisher - State Assessment Administrator, Florida 

Association for Supervision and Curriculum Development 

Alice Houston - Assistant Superintendent, Seattle Public Schools, Washington 

Council for American Private Education and National Association of 
Independent Schools 

Glenn Bracht - Director, American Lutheran Church, Minnesota 

Council of Chief State School Officers 

Richard A. Boyd - Superintendent of State Department of 
Education, Mississippi 

Council of the Great City Schools 

Lillian Bama - Superintendent, Albuquerque, New Mexico 

Dkectors of Research and Evaluation 

Glenn Ligon - Director, Department Management/Information Service, 
Austin Independent School District, Texas 

National Association of Elementary School Principals 

C. June Knight - Principal, Hobart Middle School, Oklahoma 

National Association of Secondary School Principals 

Stephen Lee - Principal, Southwood. High School, Indiana 

National Association of State Boards of Education 

Barbara Roberts Mason - President, State Board of Education, Michigan 

National Association of Test Dkectors 

Paul LeMahieu - Director, Office of Research, Testing and 
Evaluation, Pittsburgh Public Schools, Pennsylvania 
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APPENDIX A (Cont'd) 



National Community on Catholic Education Association 

Maiy Brian Costello - Superintendent, Archdiocese of Chicago, nimois 

National Council of State Legislators 

Wilhelmina Delco - Congresswoman, Texas House of 
Representatives, Texas 

National Education Association 

Robert Astrup - President, Minnesota Education Association, Minnesota 

National School Board Association 

William M. Soult - Director, St. Urain VaUey Board of Education, Colorado 

National Governors' Association 

Nancy DiLaura - Education Assistant, State House, Indiana 
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APPENDIX B 

ANALYSIS AND REPORTS COMMITTEE 

Bandy, Irene - Asst. Superintendent, Columbus, Ohio 

Burstein, Leigh - Professor, UCLA, Los Angeles, California 

Cohen, Michael - A .see. Director, National Governors' Association, 
Washington, D.C. 

Ford, Valeria - Director, Student Assessment, District of Columbia Schools, 
Washington, D.C. 

Forgione, Jr., Pascal D. - Chief, Office of Research and Evaluation, 
State Department of Education, Hartford, Connecticut 

Hess, Anae C. - Coordinator of Student Assessment, Department of Education, 
Montgomery, Alabama 

Hoover, H. D. - Professor/Director In Basic Skills Program, 
University of Iowa, Iowa City, Iowa 

LeMahieu, Paul - Director, Office of Research, Testing and Evaluation, 
Pittsburgh, Pennsylvania 

Marsden, Margaret - Member, Virginia Board of Education, 
Arlington, Virginia 

Sandifer, Paul - Director, OfJBce of Research, State Department of Education, 
Co' Jmbia, South Carolina 
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APPENDIX C 

STATE EDUCATION AGENCY CONSULTANTS 

Bemknopt Stan - Director, State Assessment Program, Georgia 

Badger, Elizabeth - Office of Planning, Research & Evaluation, Massachusetts 

Blust, Ross S. - Actmg Chief, Division of Education Testing & Evaluation, 
Pennsylvania 

Brown, William J. Jr. - Assistant State Superintendent, Research and Testing, 
North Carolina 

Chin-Chance, Selvin - Administrator, Test Development Section, Hawaii 

Clausen, Thomas G. - Superintendent of Education, Louisiana 

Cooperman, Saul - Commissioner of Education, New Jersey 

Crowe, Kevin - Director of Planning, Research & Evaluation, Nevada 

Danzig, Arnold - Director, Education Information Center, Arizona 

Davis, S. John - Superintendent of Public Instruction, Vu-ginia 

Donahue, Dennis - Assistant Superintendent, Wyoming 

Earhart, J. Troy - Commissioner of Education, Rhode Island 

Donovan, David - Assistant Superintendent for Technical Assistance and Evaluation, 
Michigan 

Fjelstad, Donna - Assistant to the State Superintendent, Division of Education, 
South DaJcota 

Gabrys, Robert E. - Chiet Program Assessment, Evaluation, and Instructional 
Support Branch, Maryland 

Grover, Herbert J. - State Superintendent of Public Instruction, Wisconsin 

Harrison, Ann E. - Director of P: jgram Planning, Research, & Evaluation Section, 
Kansas 

Hartman, Allan - Office of Planning, Research & Evaluation, Massachusetts 
Hawks, Gary - Interim Superintendent of Public Instruction, Michigan 
Hudson, Roger - Administrator, Curriculum Services, Nebraska 
Jackson, Roberta - Minimum Competency Specialist, Kansas 
Kefine, William B. - Superintendent, Delaware 
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APPENDIX C(Cont»d) 

Keister, Sally - Coordinator, Guidance/Assessment & Evaluation, Idaho 

Kerins, Tom - Manager, Student Assessment Section, Illinois 

Law, Alex - Director, Division of Program Evaluation & Research, California 

Lepley, William - Director, Department of Education, Iowa 

Lockwood, Robert E. - Coordinator, Office of Research, Alabama 

Maxcy, Horace P. - Associate Director, Division of Educational Assessment, Maine 

Moak, Lynn M. - Deputy Commissioner for Research and Information, Texas 

Randall, William T. - Commissioner of Education, Colorado 

Randall, Ruth E. - Commissioner of Education, Minnesota 

Roeber, Edward - Supervisor, Educational Assessment Program, Michigan 

Sandifer, Paul D. - Dkector, Office of Research, South Carolina 

Sansted, Wayne G. - Superintendent of Public Instruction, North Dakota 

Saterfiel, Thomas H. - Deputy State Superintendent, Mississippi 

Sobol, Thomas - Commissioner of Education, New York 

Snodgrass, H. M. - Deputy Associate Superintendent, Kenmclgr 

Strange, William B - Senior Officer, Center of School Assessment, Indiana 

Trent, E. Roger - Dkector, Division of Educational Services, Ohio 

Van Fleet, Donald S. - Director, Division of Public Schools, Florida 
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APPENDIX D 

OTHER CONSULTANTS 

Apt, Denise C. - Chairman^ House Committee on Education, Kansas 
House of Representatives 

Barrett, Ben F. • Associate Director, Kansas Legislative Research Department 

Bell, Qarence E. • State Senator, Arkansas State Senate 

Faatz, Jeanne • State Representative, Colorado House of Representatives 

Fuhrman, Susan - Professor, Rutgers University 

Godshall, Deb - Senior Analyst, Colorado Legislative Council 

Guthrie, James W. • Professor of Education/PACE Director, University of 
California at Berkely 

Hof^ Philip H. - State Senator, Chair-Senate Education Committee, 
Vermont State Senate 

Houston, Paul - Superintendent, Tucson Schools 

Katz, Vera • Speaker of the House, Oregon House of Representatives 

Keppel, Francis - Senior Lecturer on Education, Harvard University 

Linn, Robert L • Co-Director, UCLA Center for Research on Evaluations 
Standards and Students Testing/Professor, University of Colorado 

Malone, David - State Senator, Arizona State Senate 

Musick, Mark D. - Vice-President, Southern Regional Education Board 

Oakes, Jeannie - Social Scientist, The RAND Corporation 

Ollie, Art - State Representative, Chair House Education Committee, 
Iowa House of Representatives 

Payzant, Thomas W. - Superintendent, San Diego Schools 

Perry, Marda D. - Assistant Director, Georgia Assessment Project, 
Georgia State University 

Powell, Lee Etta - Superintendent, Cincinnati Schools 

SUver, Edward, A. - Professor of Mathematics Education, University of Pittsburgh 

Snyder, Cooper • Senate Chauman, Education Committee, Ohio State Senate 

Whiteneck, William D. - Chief Consultant, Senate Education Committee, 
CaUfomia State Senate 
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Ai?PENDIXE 

CURRENT NAEP PROCEDURES FOR EXCLUDING STUDENTS 



Sampling. NAEP Currently uses a stratified, three-stage national probability 
sample design. In the first stage of sampling, the United States is divided into 
geographical imits comprised of coimties or groups of contiguous counties which meet a 
minimum school enrollment size. These imits are stratified by both region and size of 
conimuni^; In the second stage of sampling, all public, parochial, private. Bureau of 
Indian Anairs, and Department of Defense schools arc listed according to three 
grade/age groups withm each primary sampling unit They are selected with a 
probability proportional to assigned measures of size. Schools that are designated as 
being only for students with handicaps are eliminated from the sample. Li methird 
stage of sampling, a consolidated list of aU grade and age eligible students is established 
for each selected school This list includes all students regardless of En^h proficiency 
or handicap. 

Exclusions. After a list of all students who are grade or age eligible is prepared, 
the directions from NAEP to local school coordinators of the assessment indicate that: 

the school coordinator should review the list of eligible students and, in concert 
with other school officials, determine whether any student should be excluded 
from assessment because thev are non-English speaking, educable mentally 
retarded, or functionally disabled...These categories are defined as follows: 

o Non-English Speaking Students - Those who do not read or speak English 
and woiiid be unable to overcome the language barrier in the test 
situation* 

0 Educable Mentally Retar ded CEMR) - Students who have been 

psychologically tested as EMR students or students who are considered 
EMR m the professional opinion of the principal or other qualified staff 
members. However, students should not be excluded because of poor 
academic tjerformance or normal discipline problem^s. Only those 
students should be excluded who could not give meaningful responses to 
exercises at their age level. 

0 Functionally Disabled - (temporary or permanent physical disability) - 
Students who are so disaolea that they cannot periorm in the NAEF 
testmg situation, should also be excluded. However, functionally disabled 
students who can respond should be included. 
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